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Moderator: Welcome to this AVEPDA session in which we will 
TOTHE EDUCATIONAL RESOURCES report on an Accountability System- I am Cal Cotrell, a 
INFORMATION CENTER (ERicr retired university professor and now part-time consultant in 
professional personnel development . 

During the 6 month period January - June 30 o-F this year, I 

had a very special and delightful experience working with a 
qq Task Force of 10 state agency and university personnel i n 

Pennsylvania- This was the Task Force for the state 
05 financed project to develop an accountability system for the 

7^ Centers for Vocational Professional Personnel Development in 

^ Pennsylvania. This project was the smallest one in dollars 

I can ever recall being responsible for, but it was most 

rewarding to me as the consultant/project director and it 
JjJJj has great potential for the continued improvement and 

funding of vocational professional personnel development in 

Pennsylvania. 

Here today to share the accountability system with you are 
the director of vocational and adult education - Jackie 
Cullen and representatives of three of the four Centers in 
Pennsylvania - Tom O'Brien, Director of the Center from 
Indiana University of Pennsylvania; Ed Herr, Director of the 
Center at Pennsylvania State University; and Tom Walker Co- 
Director of the Center at Temple University. 

Why might this presentation on an accountability system be 
valuable to you? 

If you are a t eac he r e due at o r uncertain about future funding 
from the state agency or within the university, you may f i nd 
a way to be more assured that the services you are rendering 
are better understood and appreci ated. 

If you are a state agency person concerned about how y-»u are 
going to continue to convince your superiors to fund certain 
college or university teacher education programs, you may 
\ obtain some valuable ideas from these presentations. 

(\ The accountability system has been designed for 

^ Pennsylvania, but its various concepts and features are 

J adaptable to any state or university prof ess ional personnoi 

development situation. The intent of the system is to 
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improve vocational education professional personnel 
development as well as to improve the accountability system 
itself. 

One of our AV'^PDA past presidents, Rupert Evans, was one of 
three outside reviewers for this project. It was Rupert who 
suggested that I contact Joyce Beach about this presentation 
to share the work of the Pennsylvania Task Force in tht 
development of an accountability system. 

We will be using an approach to this presentation which is, 
perhaps., a little bit different. I am going to be a 
moderator/interviewer, if you please, asking appropriate 
leading questions that will be answered by various members 
of the panel. First off, Jackie Cullen will be responding 
primarily to the rationale and context kinds of questions 
that we have about this whole system. Secondly, Tom Walker 
will have primary responsibility for presenting the 
procedures in the development of the system. Then, we will 
have Ed Herr present some of the products or components of 
the system and, finally, Tom O'Brien will be primarily 
responsible for questions that are relative to the 
implementation of the system. So, we have the story all the 
way from the beginning to the end. I am sure you are all 
concerned about implementation and I can assure you that 
these folks from Pennsylvania are going to have much work to 
do in that area. In reference to the presenters, you 
noticed that I mentioned "primary" responsibility for each 
of these four areas. That does not mean that other panel 
members cannot respond to a question that a particular 
member has been asked, if you can offer additional 
information which may be helpful to those in the audience. 
We have the potential for a state agency view, a university 
view and then perhaps since we have three distinct 
geographic regions of Pennsylvania represented here by the 
Center directors, you may get some information relative to 
differences among the three regions of Pennsylvania. After 
the panel has presented the context and rationale, the 
procedures for developing the product, and the 
implementation of this accountability system, there will be 
some time for you, the audience, to ask some questions of 
the panel. Perhaps you will have questions that may pertain 
to particular types of programs they may be offering in 
these Centers. I am sure the panel is ready to begin. 
Let's get underway with the questions. Jackie, will you 
give us a brief historical perspective citing the need for 
developing this accountability system? 

JACKIE CULLEN; Let me go back to how we got to the Center 
concept as an example. The same kinds of questions that 
resulted in the development of this accountability system 
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really are what caused us to implement the Center concept 
original ly. In 1973, our Secretary of Education at that 
time, became very concerned about how we were funding 
vocational education personnel development in Pennsylvania 
and whether we were f undi ng basic programs or were we usi ng 
Federal funds to fund over and above the normal teacher 
education programs. His opinion was that we were funding 
the basic ^"irograms- Basically;, at that time we were funding 
twenty— eight different teacher education inst i tut ions. The 
funds were going to the institutions, but there was no 
accountability system- The funds were being used to support 
the basic vocational teacher education program. He saw nc 
reason to give special funding to vocational education if it 
was runni ng the same type of programs as the rest of teacher 
education. His plan was simply to reduce the funding by a 
third for three years unt i 1 it disappeared. Fortunately, we 
were able to convince him and later Secretaries of Education 
that his p Ian was not appropr iate. A task force was formed 
that had Department of Education staff along with teacher 
educators and representatives of education professional 
associations. Their charge was to do three things: firstj 
to provide extensive rationale for administering federal and 
state funds for vocational education personnel development; 
second, to design funding strategy that would achieve the 
wisest use of funding resources; and third, to achieve a 
higher degree of accountabi lity by rec ipients of basic 
f undi ng support . In 1977, the development of the Center 
concept was completed. It was implemented in 1978 and you 
will be hearing more about the Center concept today. But, 
basical ly it provided for a critical mass concept. Those 
institutions that offered at least two undergraduate degree 
programs in vocational teacher education would be eligible 
to apply for funding in an initial contract with the 
department to provide services over and above the basic 
program. You will be hearing more about those services 
later. But, it was basically to provide as many as thirteen 
spec if ic serv ices. Four i nst i tut i ons met the cr iter ia and 
were funded at that t ime. Centers were establ ished at the 
University of Pittsburgh, Pennsylvania State University, 
Indiana University of Pennsylvania and Temple University. 
You can see that they are geographically distributed across 
the state. Then, at that time, along with the Center 

f undi ng and t he contract each of the Centers signed with the 
Department there was a reporting mechanism in place. But, 
over time the system has continued to grow and change and 
become different at each Center. Ly 1983 or so each Center 
was submitt ing a very, very thick annual report document i ng 
what was done. But the data were different from each Center 
and we were not able to combi ne it i n a report that tal ked 
about what was happening to teacher education in 
Pennsylvania. Once again, the Secretary of Education 
quest ioned "are we gett i ng the bang for the buck for our 
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teacher education dollars?" "Is what is happening at the 
Centers having a positive impact on vocational teachers?" 
Also, following that was a time when national reports were 
coming out on academic excellence followed by the Commission 
on Vocational Education Report. 1984 saw the Perkins Act. 
So, all of these things led us to, once again, examine the 
Center concept, using an outside group and PDE staff, we 
reaffirmed and issued a new position paper in 1985. And, at 
that time we also had several studies going on examining 
teacher education in Pennsylvania. Working with the Centers 
throughout this process we came to the realisation that we 
needed to sit down and start from scratch in developing an 
accountability system that would satisfy PDE as well as the 
Centers. 

MO DERATOR: You answered several questions that I hoped to 
ask. However, what are we talking about in terms of dollars 
that have been going into these four Centers? 

JACKIE CULLEN; A little over three million dollars. 

MODERATOR; Jackie, what specifically is it that your Bureau 
in the PDE hoped to achieve with this accountability system? 

JACKIE CULLEN: Several things. First of all the 

accountability system would provide data for decision-making 
at the state level. Also, the system would allow us to 
defend the system using up-to-date data. When we have new 
leaders the same questions are asked every three or four 
years. I think, also, it provides a mechanism that would 
allow for the Centers to identify areas where they need 
improvement. 

MODERATOR; What do you see, Jackie as changes in the mode 
of operation for funding the Centers that might be made as a 
result of the implementation of this new system of 
accountabi 1 ity*^ 

JACKIE CULLEN: I believe it is something that will evolve 
over time. For the first time I believe we will have 
consistent data across Centej^s and the Centers will be able 
to look at what the other Centers have done. For example 
what the costs are, in terms of costs per participant. I 
believe we will also be able to look at the effectiveness of 
activities from the current year, to help plan next year's 
activities. Perhaps it is a little early to say exactly 
what impact it will have on Center funding. I hope it will 
provide us with a defense for asking for increased funding. 

MODERATOR; I like your statement relative to your hopes 
that the accountability system will be a source of helping 
you to seek additional funds. What do those of you from the 
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three university centers -Feel might evolve in terms of 
change of operation of the Centers as a result of this new 
accountabi 1 i ty system? 

TQM O'BRIEN; I think we will see a closer relationship 
between activities that are delivered out of the 
universities to the field in terms of meeting the criteria; 
consequently, we will see more commonalities between the 
four Centers- When teachers transfer from one area to 
another that there will not be a loss of program credits in 
meeting their certification and degree requirements. The 
new accountability system will continue to help to level out 
the programs so there is less inconsistency. Such an 
arrangement does encourage the sharing of materials and 
procedures and programs between the universit ies. 

MQDERATQR; You are saying the State doesn't have all four 
Centers operating in unison at. the present time? 

TQM Q^BRIEN; No, we are not operating in unison- Each 
university has designed programs. We have shared 

information back and forth in terms of what we have designed 
and we have designed some common programs over the years- I 
think the new system will encourage greater consistency- 
Such effort will allow us to get a better handle on some 
evaluation we want to do and some research in terms of 
quality expectations that we have for vocational personnel 
preparation in the state- 

TOM WALKER; This accountability system is going to lend 
itself to developing consistency internally with personnel 
assignments and duties. At Temple, the Center is a unit 
within the college of education and faculty can contract for 
part of their assignment- Unless you have each assignment 
defined in operational terms it is very difficult to provide 
continuity internally let alone in the field. 

MQDERATQR; Ed, can you cite any other possible changes in 
Center operation? 

t.D HERR; I think one of the things that has happened at 
Penn State (or I think will happen) is a somewhat different 
approach to the typical annual report process. In the event 
that, theoretically, we go through the accountability system 
in a comprehensive manner and if we do this well it will 
reduce the typical large scale narrative that one writes as 
a yearly synopsis of achievement. In addition, the annual 
proposal writing process will be given over much more 
directly to specific data about the needs for various 
fundable activity. In other words, you will say to the 
state you intend to engage in specific fundable activities 
and you will justify these kind of commitments by needs 
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'^^^^^ .J^^ ^-^""^l proposal will be more number 
drzven more criterion related than narrative. By the 

^!::'!.,^°^^"'. -^^"^^ technical report, new written in 

2;-L:^'m ^ ^"^--^ -"-h mo;e 

ac-sp.able to cc.nouter processing, put on a disk the content 
of which IS driven by whether we have met the criteria wh^'ch 
we proposed Jor the Center for that year. I think fro^ a 
product standpoint the annual proposal and the f^na^ 

^hSv"'" /h^ ■ ''^^'y be much easier to interpret. 

They will be much more sharply defined in terms of what 
..appened, what was performed, whc was served, how .iianv 
people were served and the costs of each service categorvi 
.-.g=iin, such a xormat will allow the state and the Center= to 
be clearer about the data than is likely with the' lara-- 
narrative proposal we frequently write. 

MODERATOR = How in the world did state peopl= an^ 

university people ever get together on the collaborative 
uo tackle something as terrible as an accountability 



er-rort 
system? 



f '^L H t ' u^" preparation for this panel discussion, Cal 
talked to each or us over the phone and asked us if we wou^d 
be reeponsibile for reporting on a particular aspect of the 
project. My assignment was "project procedures." In 
thmKing about this assignment, I thought about events prior 
to getting involved with Cal and the accountability project. 
As Jackie explained, we have had Centers in Pennsylvania for 
several years. The Bureau (PDE) personnel has changed ov^.^ 
the years and so too has Center and university personnel', 
.he transience of personnel made it very difficuU to 
achieve an even implementation of the Center concept across 
our Commonwealth. For several years we have talked about 
h"!,"^ ""i^"^ improve the consistency o-f our personnel 
development services provided by Centers. During the 
semester prior to getting involved with this project, there 
were several discussions at Center directors' oteetings that 
talked to the need for "formative evaluation," for putt^na 
in place a system that would in fact get at improving" 
personnel development systems in the State. I believP that 
during that semester we began to build some of the bridges 
of trust that are required to implement a decentral i r-d 
state-wide program. The Bureau of Vocational Education in 
our state department of education is interested in helping 
the universities to put together a system that will collect 
-he information to improve Centers. 

MODiEmm Tom, will you please tell us how the task for-e 
was structured? " 

TOM WALKER There were five representatives from the four 
Centers and five representatives from the Bureau o-F 
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Vccaticnal Edu.c = ticn- It just so hacpened four of the five 
Bureau psrsonnei had recently become actively involved with 
the Centers. 



^.CpET^f^'rnp.^^ Then, what dc you regard as the main ccjmpcnent 
of this =vstsfn? 



10!!! WALTER! It finally came to the point where the task 

force decided that the Bureau should contract with a person 
or persons from o-tsids Pennsylvania to orovide the services 
that we^e necessary. Because cf cur history, ws felt we 
should look for someone who was versed in oe'^scnnel 
develocment issues, someone who anderstocd the Centsr model, 
and someone v^ho was familiar with Pennsvl vania. We all had 
an opportunity to identify individuals who we felt might fit 
the criteria. Each Center submitted namss to the Bureau!^ 
Then, the Bureau reviewed our recommendations and contacted 
and interviewed individuals and finai:y offered the contract 
to Cal. At that point, as I understand it, Cal asked for 
background information. So, we loaded an UPS truck and sent 
him several years worth of reports and evaluations. 

MODERATOR; Yes, I recall that I got more information than I 
really wanted! 

TQM WALKER: It mast have been. But, anyway, Cal was going 
to move through a process of taking a look at all of the 
background information, trying to glean from it the many 
characteristics of the system. Cal ultimately identified 
thirteen possible functions that Centers were to perform. 
What you need to know is that., historically, we had 
something like 16 or 17 possible functions to perform. 
Centers, baseo upon needs data, were able to contract for 
various components of the system. For example. Temple may 
have had 15 or 14 functions, Incxana may have had 16 or 12 — 
whatever a Center could justify with needs data would 
translate into some activity for a particular function- 
Over the years, some of those functions were collapsed or 
altered. As I said earlier, Cal's analysis ultimately 
identified 13 functions that seemed to be getting the most 
attention in our state. 

MODEPATGR ; These 13 functions were converted to possible 
objectives for the Centers. As you deal with those 
objectives, would you please share with us an outline of how 
the Task Force functioned? 

™ WALKER; As I recall, the first meeting of the Task 

Force involved putting some pretty broad parameters around 
the project itself and defining how the evaluation should 
proceed. We truly wanted it to be collaborative, we truly 
wanted it to be outcome oriented. We also wanted it to be 
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sensitive to the issues. We were concerned about making 
sure the Bureau, and the universities, had equal stakes 
invested in the total process. We were very sensitive about 
defining all of that. I might add that Cal came to the 
first meeting having done some background research on 
evaluation models. After he heard us talk about what we 
wanted and heard us describe the kind of affective 
components we wanted in the system, he suggested a 
Stuff lebeam model: An evaluation model that would consider 
context, while considering input, process, and outcome. 
Dnce we had defined the evaluation aspect we actually worked 
on one of the Center activities with the hope of outlining a 
prototype. v^Je worked a fundable activity into a performance 
ob ject ive and began to move it through a process that 
analyzed it from the standpoint of performance indicators, 
suggested evidence, and an associated rating scale. We 
took one activity that we thought all of the Centers had a 
great deal of experience with and one that operated fairly 
consistently across the entire state. At the same time we 
had to be sensitive to the Bursau because the Bureau kept us 
f ocusRd on what they were goi ng to need from the system. 
They not only were going to need to know how each of these 
activities would manifest itself at a "Center" institution, 
but what development process, performance indicators, 
evidence and cost/service record data were agreed upon at an 
institution- And this is the Service Record. 

MODERATOR: Do you want to share with us anything about how 
these initial performance objectives and cost/service 
records were developed? 

TOM WALKER: This was a process of give and take — each 
Center task force member took a turn at identifying the 
characteristics that best described the system at his or her 
i nst i tut i on- We agreed on some common performance mater i al . 
It took us all day that first meeting just to move one 
activity through the analysis process. Because it was a 
give and take situatiofi and, once again, because there were 
new people from the universities and Bureau at the meeting, 
there was a lot of educating goinn on. 

MODERATOR: In addition to the development that followed for 
each of the ob ject ives, there had to be some system 
operating in terms of getting reviews. Would you briefly 
descr ibe that for us, Tom? 

TOM WALKER; After we had the model in place Cal returned to 
his home. We had negotiated that by our next meeting we 
would like to see six activities developed to the same point 
as the prototype activity- So, Cal went back to Florida and 
started the process of doing a content analysis of the 
reports and evaluations we had sent him. He gleaned from 
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the reports enough information for \11 categories. Then, he 
started a dialogue v^ith each of th^ Task Force members by 
mai 1 . There were several passes of i nf or mat ion to gain 
agreement on the wording for some of the objectives and the 
particular cost/service data that were needed. It was 
getting to the point where we were ready for our next 
meet i ng . We met agai n in Har r isburg as a group and did some 
fine tuning. Cal, again went back to Florida and did seven 
more activities/functions. The same process continued until 
we completed all of the objectives. 

MODERATOR; I think we are ready to move i nto the product of 
the system. Ed Herr is going to carry the ball with this 
part. In terms of the product, how would you describe the 
basic structure of tfie system, Ed? 

ED HERR; Basically, it reflects an accountability system. 
It has criterion referenced performance objectives, which 
have been developed or will be developed from the fundable 
activities of the Centers themselves. And the conceptual 
structure can be described as a context input 
process/product model- The input is represented by the 
funding of Center activities. The fundable activities 
define both process and product. The processes of the 
Centers are described by the performance objectives for each 
of the fundable activities. I suggest that on the one hand 
one product of the accountability system is the redefined 
and streaml i ned proposal for the funding and performance of 
each Center and another product is the annual technical 
report of the performance. The other set of products really 
has to do with the attainment of criteria set forth in the 
performance record and the data reported in the cost/service 
record for each Center. If you will, the performance record 
and the cost/service records summarize the products of the 
fundable activities, which are then reflected in the annual 
report. They are the performance connection from the annual 
proposal to the annual report. And, they give, as Tom has 
just suggested, on the one hand criteria to document the 
performance of the Center and performance indicators that 
one can use to study one's fundable activity through a 
series of stages, steps and requirements to make sure that 
one has met the criteria for a given fundable activity. 
Rel ated to the performance record of the fundable act i vi t ies 
is the cost/service record which gives the cost per function 
per person served or per fundable act i vi ty . 

MODERATOR; You are going to refer to some illustrations in 
a moment. How about having the audience refer to page three 
of the handout. There is actually a better listing than we 
have on transparencies. 



ERIC 



10 



ED HERR; What we have on page 3 of the handout (Appendix) 
is a more sp<?ci-Fic listing of the fundable activities of 
Centers. Number 11 is "provide assistance for vocational 
educators serving handicapped learners." That can be done 
in a variety of ways. For example, one might be concerned 
about the needs of disadvantaged learners, developing 
strategy by which they can work effectively. Disadvantaged 
learners as a Center goal might be addressed in a CBTE 
format, workshop format, whatever the case might be. It 
depends on what you want to do. Number twelve in the 
handout — "Provide assistance for vocational educators 
serving handicapped learners." Again, the same dimension. 
Number one on the list "Provide center management for 
vocational professional development", which really suggests 
that each Center has a variety of components, a variety of 
fundable activities, and a variety staff, outreach staff, 
primarily at the universities but also located at other 
places. Thus, the management system, its record keeping and 
documentation, becomes very important to keep the Center, in 
all its dimensions roving in the appropriate direction. We 
have so many different parts in a Center and many different 
people contribute their performance. Number two, or page 3 
of the handout, is concerned with maintaining an 
occupational competency assessment program. Number 10 on 
the list is "Assist schools in using the Occupational Data 
Analysis System (ODAS) to develop and revise curricula," 
Each of these fundable activities become converted into 
performance objectives and each of them has a variety of 
stages of performance indicators by which they have been or 
might be met. Turn to page 4 of your handout, please. What 
you see there is an example of just one of the objectives - 
"Provide outreach services (Vocational Inter n/ I nstructor 
certification program) for personnel off-campus through 
Field Based-Competency Based Teacher Education (FB-CBTE) " 
which is really 3 on page 3. In the listing of objectives 
this would be a performance record sample for item 3 of your 
fundable objectives on page 3. So, if you turn to Page 4 
and 5, what you really see is the conversion of that 
objective, that particular fundable activity, into the 
performance indicators, which represent, if you will, the 
chronological stages, the tasks which are required to 
achieve that broad objective. The second column represents 
suggested evidence of the performance indicators and, of 
course, the rating scale. The evidence can vary somewhat, 
depending upon the Center's own staffing and how it is 
engaged in the process. That is why the form lists only 
suggested evidence. The performance indicators tend to be 
more standardized. The evidence as indicated may vary 
somewhat, depending on the indigenous characteristics c^t a 
particular center. Then, on page 6 of the handout, you see 
the Cost/Service Record. In the Cost/Service Record, what 
you are really trying to do is to determine how much it 
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costs to deliver each fundable activity at a particular 
Center and across the state; the cost/service record permits 
the state bureau to compute an average of how much money is 
being put into each fundable activity out of the three 
million dollars Jackie mentioned earlier. ^or example, how 
much of that money is going to provide adequate services to 
personnel off campus in field based-competency based teacher 
education? How much is going for occupational competency 
assessment? How much for ODAS, etc.? These data, then, can 
be computed both in terms of the Center and in terms of the 
average for the state. So, what you have here is both a 
cost dimension per objective and a service dimension- One 
can determine how many people actually have been served in 
the activity, what ^ind of competencies they are completing, 
the number of competencies they are completing and the 
number of competencies that have been completed. Obviously, 
the actual service would depend upon what the fundable 
activity would be. In some cases you are talking about 
workshops, not courses. In other cases you are talking 
about the schools served rather than individual clients- 
Or, the number of certificates being issued or recommended 
to the Pennsylvania Department of Education- So, again, the 
Service Record has to be defined in terms of what is 
required by the performance indicator and the fundable 
activity. The service dimension could vary somewhat- But, 
as you can see, you have an awful lot of comparative 
possibilities — those within a Center or across Centers, 
but certainly in aggregate you can give some very detailed 
information to policy makers and state legislators and to 
others, about what the Centers are doing in specific 
pf?rsonnel development terms. Obviously, it is quite 
possible that the current 13 fundable center activities or 
objectives will be modified as the personnel development 
needs of the state require additional fundable activities to 
be added or, in fact, it may well be that we will get to the 
point that certain eiiisting fundable activities will not be 
nee Jed and, therefore, there is built into the system, at 
least conceptually, the opportunity to modify these 
objectives so that the Center^s can 'meet the state needs in a 
very systematic way. In that sense the cost/service record 
becomes a product which influences what is written in the 
final technical report. The assumption is that this 
material, this inFormation, can be put on floppy disk, can 
be transmitted to the state in those forms to allow the 
state to much more economically and rapidly assimilate this 
information into their total data and planning bases. 

MODERATOR; I believe, Ed, you indicated the Task Force came 
up with around 19 different itp;ns for the cost record and 53 
or 54 different items in the service record. Would you care 
to lead off on any comments relative to what some o-h the 
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fears and anxieties miyht be of this system? It supposedly 
has b*en accepted by all four of the universities 
and by the Bureau of Adult and Vocational Education, but I 
am sure there might be some concerns. 

ES tiERRl I think there are. I do not know that the 

concerns are rampant- But, nevertheless, there ars always 
concerns- I think that, as Cal has said, putting this 
system together requires a lot of tru<^t. You have to assume 
that those in the state and your colleagues in other Centers 
are playing it quite straight that you are all in this 
together, that we are a collective enterprise tr/ing to 
support each other. If you do not start with that 
assumption you have a very real probltm of honest 
presentation of data. But, the fnars, I suspect, are how 
will this comparative data, by function, by Center be used. 
It is assumed that there are all kinds of ways to aggregate 
this data and compare it. So, ultimately, one has to say 
that it may well be cheaper to do a particular activity at a 
particular Center than another Center. What does that mean 
to thE' C^-^nter which is more expensive? What does that oiean 
in ter nc; of state judgement about these ^ata? That is a 
very tricky question. Will the context of the geography of 
the Centers really be appropriately inrluded in the 
judgement about funding, explaining why some Centers may be 
more ejrpensive than othsr Centers. To give you an example, 
the Centers really operate in three major service districts. 
The Center which I represent at Penn otate Uni> ersity is by 
truism and maxim, I gues^b, equally inaccessible from 
everywhere, which is to say we have a very large Center, a 
very large university, in the middle of a whole lot of 
trees, lots of deer and bear, etc. And, so our area as 
compared to Tom's area is less densely populated and more 
geographically dispersed. But, the point is that Temple is 
located in basically an urban area. Obviously there are 
rural areas and semi-rural areas in their service area, too. 
But, there are more and larger concentrations of people and, 
therefore, the Temple Center delivery system is somewhat 
differently configured than are those of Penn State where, 
as I said, we have a lot of deer and bear as well as very 
few people for long stretches of our service area. So, we 
have to figure out how to deal with some of these issues. 
They may be farther apart than we think they are. We don't 
know. But, these data on costs of services, I think, will 
be helpful. Obviously, there are concerns such as, will 
cost/service differences heighten the competition for funds 
among the Centers? If that happens in a very dramatic way 
trust relations will be a little bit harder to maintain. In 
terms of some of the performance indicators that we have 
relative to competencies; how many competencies do certain 
people require when being served in field based programs or 
workshops^ Again, what are the competencies one might 
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expect? How do you put numbers on some of those kinds o-F 
things'^ 



MO DERATOR; I believe one o-F the problems there, Ed, is that 
while each university has filed for and received program 
approval by the state agency they may not end up with all of 
th€* same competencies for a given program. This problem 
calls for the state to establish a set of minimLim standai ds 
or number of spec if i ed competenc ies for each program to 
receive approval . 

ED HERR: I think there are a number of fea""s and anxieties 
yet to be pi ayed out in terms of ex pec tat ions about such 
competencies. I don't have them personally, but they are 
rather apparent in this kind of accountabi 1 i ty. 

MODERATOR: Any of the other regions wish to respond to the 
same quest ion regardi ng fears and anx iet ies? Maybe Jackie 
might have some fears or anxieties relative to the State 
Director' s posit ion. 

JACKIE CULLEN; I guess I would react to that statement 
differently for different kinds of universities. One is a 
Land Grant university, one is part of the state system of 
higher educat ion and two are state related. Because of the 
nature of the different kinds of universities, there are 
different costs, and differcrnt administrative structures. 
There are different costs of servicing or delivering any one 
activity based upon the institution- I have concerns as to 
how to use tfie cost data. Obviously, when you are dealing 
with -hour different institutions you have institutional 
policies you need to take into consideration. An additional 
concern iG> the amount o^ time involved in data reporting. 

.T.ast not allow it to grow cumbersome and time consuming 
for the Centers. 

MODERATOR; I am happy to learn you have thrown away the 
scales for weighi ng those proposals and reports. Jack ie. 

TOM WALKER: I think the group, at our l^st meeting, 
expressed some concern about the perpetual descr ipt ion that 
Ed referred to and, as I recall, we agreed that the system 
would not be fully developed until we brought it to that 
particular point- I believe that was part of our charge as 
we moved into the next plannir»q cycle- We tried to create 
the discussion around t he context so that we coul d u^e 
context as part of the accountability system. 

MODERATOR; As a matter of fact, one of the recommendations 
coming out in the final report had to do with getting at 
context. Now, I believe we are ready to get a little mor^^ 
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detail on implementation, Tom, would you share that with 
us? 



TQM O^BRTENl On page 7 of the handout (Appendix) you may 
observe time sequence that we go through* as Centers in 
working up our materials to be Bent to the Bureau for 
consideration for funding. It starts with July 1 being our 
Christmas. We receive money on that date for that year. 
So, we get a Christmas in July in Pennsylvania. The review 
meetings and process tha^ you see beyond that, indicate 
there is a little more detail in terms of what goes on 
during the year related to Centers and our interaction with 
the Bureau and with other Centers as we plan activities 
across the state. You'll notice there are a number of 
meetings with state agency personnel as we either go into 
their offices or they are out to visit with us at the 
universities and to spend time with the Centers and also 
with university administration, if you would, where the 
Centers are housed. As you look at the list of Center 
objectives (fundable activities) please notice most of them 
are non-traditional types of activities that are delivered 
through the Center, delivered in the field based delivery 
system, which means they are off-campus in a lot of 
situations. Some of them are unique services that have a 
tendency io parallel other services the university may 
provide in terms of placement, student record keeping, 
accountability. So, there is a continuous process going on 
in keeping university people knowledgeable of what a Center 
is doinq and why we are doing certain activities. As you 
loo^ at this there is a chance?, for example, for us to go 
into a midyear review in January, which again is part of our 
evaluation process, in terms of formative evaluation, we are 
looking at what is going on in the fM'st half of the year 
and how we might adjust that or what additional data we 
would need to collect in order to verify that we are 
providing services for which we have been funded. So, this 
is a continuing process. What you don't see up here is a 
requirement that we use advisory committees for strategic 
planning during the year to verify all of the activities 
that are goi ng on within a Center. There is a very 
elaborate^ process that brings the field into our operation 
or to the Center operation to review and make 
recommendations as to how activities will be formulated, how 
funds will be spent, and where these activities will take 
place, for eiiample, in a given region. That also gives the 
Bureau an opportunity to look at, in essence, a client 
feedback system that goes back to the teachers, the 
administrators, the school district and ask what kind of 
services have we provided. It is a double check kind of 
mechanism or safe.'ty cr.eck, if you please, in the system. It 
is not represented on this chart. 
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MODERATOR; Each one of the performance records does call 
for needs assess.Tent^ as I recall. What is the kind of 
activity that would go on for needs assessment. 

TOM O^BRIEN: In order to do an activity, we must perform a 
needs assessment to indicate if there is a critical mass of 
individuals or schools who want a particular service. We 
also must do some type of a priority of that, we have "x" 
number of dollars to spend in a particular activity against 
the 13 objectives. One activity may be emphasized in a 
given year more than in a previous year or future years. We 
then need to establish objectives to meet each of those as 
common and, as Ed indicated, look at how we are going to 
evaluate ourselves, how the field is going to evaluate them. 
Then. basically, come down to do the cost analysis that is 
related to that. 

MODERATOR: How are you going to go about collecting 
Cost/Service record data? There has been some rumor that 
you might have to hire sixteen additional people. 

TOM 0^ SRIEN: The university accounting systems do not give 
us the kind of detail that need to be verified. If you look 
at the sheets that Ed mentioned in the packet in terms of 
cost data, each Center, in essence, is required to set up 
its own accounting system. We, at lUP, have set up a system 
in- house that is operated within our office that does two 
things. It gives us this cost data and also protects us 
within the university from losing funds through their 
accounting systems, which aren't very accurate. We find 
lots of times that our soft money grants get charged 
inappropriately for expenditures that we have not incurred 
and we are constantly fighting with them to keep their 
fingers out of our money. For protection, we use an in- 
house purchasing and accountiny system that is computeri::ed. 
We find that data eritry and tracking takes a lot of time and 
we are now moving into bar code technology to put down the 
costs so it is very realistic to u »e within that system. It 
also works in terms of Eervict^s to the students, for 
example, if you take a look at f age 4 of the handout, 
performance indicators numbers 3 and 4 on this particular 
activity require record keeping -Tor performance assessment, 
etc. For this purpose, ..r ^-^/e a record keeping system 
computerised that indicates every contact with a teacher. 
This one calls for new teachers to get a contact once every 
two weeks. Again, this information is being converted to a 
bar coding entry system. We currently do it with the field 
people filling out a forr , bringing it in so it can be 
entered i nto t he computer system. Two peop le are now in the 
process of changing that over to a bar code entry system 
check-off and it makes it very simple for a work study 
student to computerise the data. We do not have to use a 

15 



16 



secretary, which keeps cutting our costs. One of the major 
issues of the Cent€:r is to continually look at processes to 
determine how we can cut costs to be more efficient. 

MODERATOR: In terms of using the performance records and 
cost /service records, I assume that you are concerned not 
only about formative evaluation of the Center and its 
function, but also about the formative evaluation of the 
account ab i 1 i ty system. 

TOM 0^ BR I EN; That is correct. The system is a very 
important factor . There is consistency in terms of data 
that is entered, how it is collected is consistent, we use 
instruments that we have used over a period of time. In 
other words, we are sure we are collecting the same types of 
data each t i me out . We do make a 1 ot of compar i sons of data 
internally. As Ed pointed out, this kind of system has a 
number of functions if you compare between Centers, that is 
the purpose. But, one of the largest uses is from the 
management standpoint internally within the Center is to 
compare what has happened to us this year over the previous 
year and what kind of modifications we need to make in order 
to operate more effectively and efficiently. 

MODERATOR; So, you use an internal evaluation process? 

TOM O'BRIEN: That is correct. 

MODERATOR; You have i ndicated by what media you i ntend to 
be communicat i ng with the state agency. Floppy d isk , 
pr imar i ly . Is that correct? 

TOM O'BRIEN: At the present time, one of the bureau staff 
members is developing a program on "d" base that will be 
u^ifd to enter our data so that we have a common format for 
entering data from the Centers which will make our lives 
easier. We can enter that on floppy disk and send the 
floppy disL with hard copy backup as compared to a lot of 
other data collection we did in the past and also that can 
be used for comparison between Centers or within Centers on 
act ivit ies- 

MODERATOR; What, in your last meeting, was the tentative 
agreement about this compar ison among the Centers i n terms 
of the cost /serv ice recor d data'' 

TOM O'' BR I EN: I bel ieve there will be some cost compar i sons 
done. However , these will be done through exam i n i ng state 
averages, etc. , as opposed to examining each indi vi dual 
Center " s d ca. We have agreed not to share i nd ivi dual 
Center reports with other Centers- Again, I think Jackie's 
concern, from the Bureau standpoi nt is important . That is, 
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the need to look at data across Centers in terms of what 
context and pol icies each of us 1 i ve witn in our systsfTi. 
What I would anticipate will come out of that is that we 
will probably move away from the cost analysis at that Itvel 
and look for at an outcome level in terms of our impact on 
professional development and in particular c lassroom 
teaching/learning activities that go on. I think as we grow 
in this process, and as we move forward in it, we will start 
looking at our impact into the educational system, more as an 
indicator of how well we are working with personnel. We 
then can start looking at outcome measurements more 
ob ject ively . 

MODERATOR; As I understand the agreement, you will not 
necessa^-^ly be sharing data from one institution to another. 
It will be up to the state office to compile, analyze and 
then give you averages, and so on. It will not be a direct 
exchange 'laying Temple did this, Indiana did this, Penn 
State did thus and so. But, you will probably each know who 
did what regardless how you go about handling it. You have 
said that you are going to use the floppy disks and 
printouts from your computer to the extent that you can 
reduce the amount of volume of your proposals as well as the 
final report. 

O'BRIEN: That is correct. I think each Center has 

taken a hard look at reduc i ng the si ze of proposals and 
requests. I would anticipate over a period of time that we 
will probably get into some standardization of the proposal 
process. In our particular case, we are using Ventura 
Publishing to print that document out each time. We have 
been on WordStar for a number of years. This proaram makes 
it easy to generate the document by simply takinn the needs 
data that we collect each year and go back through the 
document and enter the newest needs data along with last 
year's so that we ;iave a fairly decent historical track of 
what has happened in particular areas of the Center. 

MODERATOR; What is the schedule ^or implementation? When 
are you going to have the first output of the system? 

TOM O^'BRIEN: Implementation is during this academic year. 
We will be ei^amining the kinds of d^ta that will be 
collected via the data base system that is being developed 
and we will be writing our proposals for tlis coming year, 
which will be the funding of July 1989 against this system. 
And, that will be the year (July 1, 1989 to June 30, 1990) 
for which we will also collect data in compliance with the 
new accountabi 1 ity system. 

MODERATOR; The process would begin with the proposal in 
March and by July 1 you would have your Christmas in July 
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-For the following 1989-1990 school year. So, it would be 
the Fall of 1990 before you would have data to be analyzed 
at the state level. 

TQM □''BRIEN; That would be the end of the first cycle. 

MODERATOR: For the benefit of our audience, what 

anticipated advantages do you panel members see in this 
accountability system? You have mentioned various ones. 
Please comment on some of these that you have cited during 
your presentations to summarize the high points. First one, 
dealing with communication. I recall that the five members 
from the state agency really wondered why you needed all of 
this performance record stuff? What happened? 

TOM O'BRIEN: There was a discussion. I think they were 
looking to boil the data down to a smaller amount than we 
were. But, from a management standpoint of operating a 
Center within a university that does deliver non- 
traditional kinds of activities, we have, as you might 
guess, some major communication problems in-house at times 
describing what it is that ws do and why and why we have 
certain kinds of leeway in some of the operations that we 
do, etc. So, I think the people managing the Centers built 
a case saying the data we need to collect do one of two 
things. One, is the accountability that we have performed 
it. Tv^io, that it is a formative evaluation process ongoing, 
that we can make continuous changes in our system. We want 
to make sure that we are collectinq enough data that we can 
do that. 

MODERATOR; Why is it that you inention trust as being one of 
the advantages of this system? 

TOM 0^ BRIEN: I think trust became critical as the Centers 
evolved from the standpoint that we had different players 
come in and out of the system, at the state level, at the 
university level. This system gives us a common base in 
which to discuss items or activities that are going on. So, 
that we at least have a common place to come off from a 
particular activity and there is a meeting of the minds. 

MODERATOR: An/ further comment relative to evaluation, the 
self-^evaluation or external evaluat ion you see as an 
advantage to the system. 

TOM O'BRIEN: I think one of the other advantages that we 
tend to overlook is the fact that we model out of the Center 
what it is that we would like to have happen in the school 
system in terms of evaluation, in terms of self-evaluation, 
the potent i al outcome, etc. I think another sp inoff effect 
of Centers is managing personnel development as we do, is 
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that we model for the schccls, for administrators and 
tsac^srs the kind c-F bshavicr, for the kinds of acclivities, 
the performances we would lise to see in the schools. 9o, 
do h£:v£ rrits^-ia that we hol^ ourselves to. We are 
c'jmpstspoy based for perf oriiancs. We evaluate ourselves, we 
ar-B continually condLicting self-evaluation. But, I think 
thet is another spinoff that may not have been an 
intentional outcome at the beginning, that has now become an 
integrsil part of the Center process- 



mai ntai n i nc 
Center'^ 



Hew d 

SUuOC 



do ycu see 
that is 



this syststT) helping vou in 
financial sucocrt for the 



TON WALVEP; We have probablv cut across the previous two 
ends. Con^'Dunicat ions, trust and now willingness to support. 
Prior to the whole process,, if you we>-s to pick up our state 
plan for vocational education the description for personnel 
development centers consumed a few lines. A positive 
outcome of this particular project was that we now find that 
the stats plan has Centers very well defined. Jly feeling 
was that as political parties changed in Harrisburg, as 
legislative teams came in and cut, it was very easy to pick 
up the state plan and take a look at it and say, "well what 
is this three million dollars doing, what is a Center?" The 
plan did not describe the Centers very well. Each time 
there was a new Governor and a new team, it seemed we had to 
defend the Center concept. I am not so sure this won'^t 
still be the case in the future, but I believe we do have a 
better base for describing what we do and for explaining the 
expense for what we do. And, hopefully, we will have the 
service data to back it up. 

MODERATOR; Anyone else care to comment? I believe 
that maintaining support is a very important advantage that 
the Task Force realised. The panel members have talked 
about simplifying reporting. Jackie has mentioned that she 
will have better ammunition in defense of personnel 
development when she goes to the powers that be, including 
the legislature, for funds. Panel members have commented on 
the results of the accountability system being used for 
program improvement purposes. They have commented, too, on 
management. How internally it will help them in terms of 
having clearly defined responsibilities for the coordinators 
of the various functions or objectives- As they change 
personnel, it will be easier to keep track of and be sure 
that services called for are being performed in accordance 
with the plan. Obviously, all of this is done for program 
improvement- The panel members have indicated that they are 
going to be dedicated in trying to improve the 
accountability system, too, as time ►-oils along. 

1? 



Now, perhaps some of you would have questions that you would 
like to asf- the panel members. 
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ABSTRACT 



This project was originated in response to a mutual need for better 
communication between the Pennsylvania Department of Education, Bureau of 
Vocational and Adult Education (BVAE) and four universities in Pennsylvania 
operating vocational education professional personnel development Centers. 
The purpose of the project was to design an accountability system for the 
professional personnel development Centers through a cooperative effort with 
Center directors and state staff. The objectives of the project were: 
1) Develop measurable process and product objectives for each fundable Center 
activity; 2) Design a recordkeeping system with a orocedure compatible for 
computer application to provide cost and other pertinent data; 3) Design an 
instrument for collecting information from staff regarding time, travel, 
etc , pertinent to each Center activity; and 4) Design a procedure for for- 
mative and summative evaluation of Center activity which uses both internal 
and external personnel. 

The procedures for the project included the following: A Task Force 
was organized in January 1988 to coordinate the collaborative effort and 
advise the project director. During the initial meeting of the Task Force 
consisting of 10 membei s (five from each, the BVAE and the universities) the 
broad specifications for a system design were developed. A computer search 
was initiated for relevant accountability systems or models. The basic 
design for this accountability system was derived from task force delibera- 
tions. A sample for one function of Centers was outlined during the first 
Task Force meeting. The consultant/project director completed a content 
analysis of proposals and technical reports of the Centers to develop the 
performance record consisting of assessment criteria and relevant cost/ 
service items for each Center objective as a fundable activity. As materials 
were developed, they were mailed to the Task Force members for review indi- 
vidually prior to formal meetings. Recommended revisions were discussed and 
agreed upon in meetings of the total Task Force. During a meeting in March, 
the original draft of the accountability system materials was approved by the 
Task Force; also, a schedule was established for conferring with the per- 
sonnel at each of the Centers to review the system and obtain additional 
input. The meetings at each Center were completed in April and the suggested 
improvements were developed and mailed to the Task Force for review and 
approval. The system's design was evaluated by three outside reviewers as 
well as other appropriate members of the BVAE.- 

The Task Force approved the materials and schedule of yearly activities 
for the implementation of the system on June 16, 1988. Recommendations 
relative to implementation of the accountability system were developed. 
Cost/Service data will be produced during 1989-90. State-wide statistics 
will be examined for the first time in November 1990. 
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Currently Identified 
Fundable Center Objectives* 

1. Provide center management for vocational professional personnel development 

2. Develop and maintain a comprehensive Occupational Competency Assessment 
(OCA) Program 

3. Provide outreach services (Vocational Intern/instructor certification 
program) for personnel off -campus (a) Undergraduate Courses (b) FB-CBTE * 

4. Provide outreach services (vocational education curriculum specialist, 
supervisor and director certification program) for personnel off-campus 
(a) Graduate Courses (b) FB-CBLT 

5. Provide workshops and seminars for personnel from all vocational content areas 

6. Provide pre-induction, professional education for clients recruited from busi- 
ness and industry with little or no pedagogical education or experience 

7. Provide an Occupational Experience Program 

8. Provide placement services for vocational professional personnel 

9. Provide a recruitment system for vocational teachers and leadership 
personnel 

10. Assist schools in using the Occupational Data Analysis System (ODAS) to 
develop and revise curricula 

11. Provide assistance for vocational educators serving disadvantaged learners 

12. Provide assistance for vocational educators serving handicapped learners 

13. Provide suppo»*t for a research arm in vocational education 



* Order of listing does not necessarily indicate priority. 
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.PERFORMANCE RECORD 



Objective: Provide outreach services (Vocational Intern/Instructor 
certification program) for personnel off-campus through 
Field Based-Competency Based Teacher Education (FB-CBTE) 



Performance Indicators 

1. Clients' needs were 
assessed 

2. Prospective clients were 
informed of FB-CBTE 
program and requirements 



3. New teachijrs were 
observed once each week 

4. Experienced teachers were 
observed once each two 
weeks 

5. Individualized helping 
conferences were provided 

6. Opportunities were pro- 
vided for coUegial 
interaction 

7- Offerings were modular- 
ized and self -paced 

8. Clients making less than 
average progress were 
counseled 

9. Staff (resource person) 
participated in regular 
training weekly 

10. Staff engaged in 
meetings regularly 

11. Modules were appropriate 
for specific needs of 
clients served 



Suggested Evidence 



Rating Scale 



1.1 Results of surveys to 

support need cited/recorded NA Yes No 

2.1 A 25% sample of prospective 
clients was contacted to 
determine if they received 
timely information 



3.1 Documented, teachers* 
records and staff travel 

4.1 Documented, teachers' 
records and staff travel 



5.1 Documented, teachers* 
records 

6.1 Documented, teachers* 
records 



7.1 Documented, teachers* 
record of progress 

8.1 Documented/signed-off 
on teachers* records 



9.1 Record of meetings 
documented 



10.1 Documented record of 
meetings 

11.1 Documented record of 
yearly review of modules 



NA Yes No 

NA Yes No 

NA Yes No 

NA Yes No 

NA Yes No 

NA Yes No 

NA Yes No 

NA Yes No 

NA Yes No 

NA Yes No 
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Objective: Provide outreach services (Vocational Intern/Instructor 
certification program) for personnel off -campus through 
Field Based-Competency Based Teacher Education (FB-CBTE)( contd.) 



Performance Indicators 

12. Clients evaluated the 
total delivery system 
regularly 



Suggested Evidence 

12. 1 Documented record of 
yearly evaluation of 
staff, curriculum and 
delivery methods 



Rating Scale 



NA Yes No 



13. Vocational Instructional II 13.1 Documented record of 
certificated resource per- "local supervisor" review- 
sons were involved in 
helping clients served 

14. Competency evaluation 
panel functioned in each 
school 



ing competency attainment NA Yes No 
14.1 Documented teacher records NA Yes No 



15. Senior teacher educator 
supervised work of all 
field and resident 
resource persons includ- 
ing service in competency 
evaluation panel 

16. Competencies and an 
attainment scale were 
used for each certifi- 
cation level 

17. A process was in place 
for evaluating compe- 
tencies of clients 
transferring amrng 
Centers 



15.1 Documented, teachers' 
records and teacher 
educator report 



NA Yos Ho 



16 J Documented teachers* records NA Yes No 



17.1 Documented by written 
policy 



NA Yes No 
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5AMPLE 



COST/SERVICE RECORD 

Objective: Provide outreach services (Vocational Intern/Instructor 
certification program) for personnel off-campus through 
Field Based-Competency Based Teacher Education (FB-CBTE) 

Cost 

Total cost 

Cost per competency completed (TPO) 

Service 

Number of teachers served 
Number of competencies completed (TPO) 
Average number of competencies (TPO) completed per client 
Number of certificates issued/recommended to PDE: 

Voc. I 

Voc- U 

Instr. I 

Instr. II 

Number of certificate endorsements for Co-operative Education 
recommended to PDE 
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SCHEDULE OF YEARLY EVENTS FOR THh ACCOUNTABILITY SYSTEM 
July 1 state agency funding of the Centers 

■^o^^Julv H?'!^.'^'^"''-^Jo"^ °^ y^^'^ '■e^'^w "meeting with Center 

or July directors and Bureau program personnel 

Sept. 30 The Centers' fiscal and technical reports are due for 

previous year 

Nov. 1 {1){2) State agency combination review of cost/service record 
state-wide statistical summary and preliminary planning 
meeting with Center directors ^ 

Mid-Nov. (1) state agency site visits to each of the Centers 

Dec. (2) State agency planning meeting with Center directors 

state agency mailing of guidelines for funding for next 
fiscal year ^ 

(1) State agency mid-year review meeting with Center directors 

f^n!%'h'^^^"r^'"^^'°" ('"i<l-year) with assessment criteria 
from the performance record for objectives 

SSr?o"'neTnsJ:/;ea:""'^""^ --^^-^-^tion as Center 
Mid-April (1) State agency site visits to each Center 



Jan. 

Jan. 
Jan. 

March 



May 



Note: 



ERIC 



?rI!m%^'^'^T'^"'^'"°" ^^""^ °^ y^^'"' with assessment criteria 
from the performance record for objectives 

' !??iT^''j' i'^'""'^ °^ '^'^^ ^^^"'^y site visits and review meetings 
a St '.h'L"?/^'""' <<evel oping solut ions to robTem to 
P^J^saltr'^n'HinS? 

^ ItltTlirir^^A^ f 5^ '""^"9 "'^^tings will be to discuss 
de?prZffh"f'''-^°'/''°r"''°"'l personnel development and to 
?unc[iJns durlSn'^^h''^ °f l^' objectives fSr the Centers' 

needs asLtZ? ?.LV^ ll'^'^ y^''' '^^9^°"^^ ^"<1 state-wide 
nihe e S s ''''''''' 
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ROSTER OF TASK FORCE MEMBERS 



Richard Adamsky 

James Bishop 
Clarence Dittenhafer 

Edwin Herr 

Carolyn Kratz 

Ruth Lungstrum 

Thomas O'Brien 

William Reilly 

Thomas Walker 

Thomas Winters 



Director, Center for Vocational Education Professional 
Personnel Development, Temple University 

Personnel Development Coordinator, BVAE, POE 

Research Associate, Research, Evaluation and Data 
Management Section, BVAE, PDE 

Director, Center for Vocational Education Professional 
Personnel Development, Pennsylvania State University 

Head, Curriculum and Personnel Development Section, 
BVAE, PDE 

Director, Center for Vocational Education Professional 
Personnel Development, University of Pittsburgh 

Director, Center for Vocational Personnel Preparation, 
Indiana University of Perinsylvania 

Chief, Division of Vocational Education Planning, 
Development and Evaluation Services, BVAE, PDE 

Co-Director, Center for Vocational Education Profes- 
sional Personnel Development, Temple University 

Head, Research, Evaluation and Data Management Section, 
BVAE, PDE 



Calvin J. Cotrell Consultant/Project Director 
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